[EEM] Replace hashed ID with human readable ID #193652

miltonhultgren · 2024-09-21T20:23:17Z

This PR turns the entity.id field format from a hashed value to a human readable string of the values found in the identity fields, such as my_host-my_cloud_zone for the identity fields [host.name, cloud.availability_zone].
The order of the values is based on the order in the identity fields list.

obltmachine · 2024-09-21T20:23:30Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

miltonhultgren · 2024-09-21T20:23:38Z

/ci

...ck/plugins/entity_manager/server/lib/entities/ingest_pipeline/generate_history_processors.ts

x-pack/packages/kbn-entities-schema/src/schema/common.ts

dgieselaar · 2024-09-30T15:38:18Z

...ck/plugins/entity_manager/server/lib/entities/ingest_pipeline/generate_history_processors.ts

+        value: definition.identityFields
+          .map((identityField) => {
+            const field = `entity.identity.${identityField.field}`;


can we sort the identityFields to make sure it is consistent even if the order of identityFields for whatever reason changes?

the pr desc mentions this but looks like it's not implemented yet

The one problem I see with this is that any consumer of the definition must then also remember to sort the fields first before recreating the ID.

While if we rely on the order of definition, it is up to the author to ensure the order makes sense as the set of fields changes.

Could it be that it's a valid case to change the order of the fields?
It should not have any semantic meaning, but may lead to "prettier" IDs being generated depending on the values found in the used fields.

In a world where we don't do materialization (by default), this seems like a smaller problem?

We did have some discussion recently where we said that essentially, if you change the identity fields, we should just throw away any previously materialized data as well.

I think we can expose utility methods that convert the id into an entity, and given that, I would personally prefer sorting the identity fields because we can sort it always when we deserialize, instead of hoping no one changed the order of the identity fields (for whatever reason).

In a world where interactions go through an entities client, that should be fine, I'll add the sorting then.

I added an issue for this: https://github.com/orgs/elastic/projects/1504/views/1?filterQuery=id&pane=issue&itemId=82919655

kibana-ci · 2024-09-30T16:47:46Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b872225e606bf46bc9cc229110a10c9ff469e1ff
Storybooks Preview

Failed CI Steps

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`inventory`	152.2KB	152.3KB	+73.0B

History

💔 Build #236153 failed 3567c9bce6bd0149caf14f535c937d78760931f9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

miltonhultgren · 2024-10-17T14:35:34Z

x-pack/packages/kbn-entities-schema/src/schema/common.ts

@@ -145,7 +145,7 @@ export type MetadataField = z.infer<typeof metadataSchema>;
 export const identityFieldsSchema = z
  .object({
    field: z.string(),
-    optional: z.boolean(),
+    optional: z.literal(false),


This blocks people from using optional fields even though technically this is still supported in the code, and we transform the existing saved objects to remove them.

This is mainly due to not being able to change the saved object mapping.

miltonhultgren · 2024-10-17T14:36:58Z

x-pack/plugins/entity_manager/server/saved_objects/entity_definition.ts

+  EntityDefinition,
+  EntityDefinition
+> = (savedObject) => {
+  // Doing only this may break displayNameTemplates


Which I think is okay since we're likely to remove this anyway in the future since it's hard to do any kind of template in ES|QL at runtime. So the displayNameTemplate might become similar to the identity field, where we just let users decide which fields to concat instead.

… src/core/server/integration_tests/ci_checks'

elasticmachine · 2024-10-18T08:38:57Z

💚 Build Succeeded

Buildkite Build
Commit: ada614c
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-193652-ada614c40f53

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`inventory`	218.6KB	218.6KB	+2.0B

History

💔 Build #243569 failed 4bd15fa
💔 Build #243135 failed bf3fe564538a52a4d8f0fc681fa043b31f975b8f
💔 Build #242340 failed 42476eb568ee60be62d5ea6f47b0b2368610a659
💔 Build #242244 failed eddd9bdbea45d4da48d11991968e7ddacbdad0da
💔 Build #241198 failed cebdb3266197c18bc2cf966b2ce7821a74c99645

jloleysens

Unsafe transform LGTM.

kibanamachine · 2024-10-18T14:42:59Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11405848322

This PR turns the `entity.id` field format from a hashed value to a human readable string of the **values** found in the identity fields, such as `my_host-my_cloud_zone` for the identity fields `[host.name, cloud.availability_zone]`. The order of the values is based on the order in the identity fields list. --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit ae2c6ad)

kibanamachine · 2024-10-18T14:47:49Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [[EEM] Replace hashed ID with human readable ID (#193652)](#193652)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Milton Hultgren <[email protected]>

This PR turns the `entity.id` field format from a hashed value to a human readable string of the **values** found in the identity fields, such as `my_host-my_cloud_zone` for the identity fields `[host.name, cloud.availability_zone]`. The order of the values is based on the order in the identity fields list. --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit ae2c6ad)

) # Backport This will backport the following commits from `main` to `8.16`: - [EEM] Replace hashed ID with human readable ID (#193652) (ae2c6ad)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Milton Hultgren <[email protected]>

miltonhultgren added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Feature:EEM Elastic Entity Model labels Sep 21, 2024

miltonhultgren marked this pull request as ready for review September 30, 2024 14:48

miltonhultgren requested a review from a team as a code owner September 30, 2024 14:48

klacabane reviewed Sep 30, 2024

View reviewed changes

...ck/plugins/entity_manager/server/lib/entities/ingest_pipeline/generate_history_processors.ts Outdated Show resolved Hide resolved

klacabane reviewed Sep 30, 2024

View reviewed changes

x-pack/packages/kbn-entities-schema/src/schema/common.ts Outdated Show resolved Hide resolved

dgieselaar reviewed Sep 30, 2024

View reviewed changes

klacabane mentioned this pull request Oct 9, 2024

[eem] remove history transforms #193999

Merged

miltonhultgren force-pushed the eem-non-hashed-ids branch from 8cc989a to 9c3997d Compare October 10, 2024 13:01

miltonhultgren requested a review from a team as a code owner October 14, 2024 15:16

botelastic bot added the ci:project-deploy-observability Create an Observability project label Oct 14, 2024

miltonhultgren requested review from a team as code owners October 14, 2024 19:30

miltonhultgren requested a review from jaredburgettelastic October 14, 2024 19:30

[EEM] Replace hashed ID with human readable ID

9afd5ad

miltonhultgren force-pushed the eem-non-hashed-ids branch from bf3fe56 to 9afd5ad Compare October 17, 2024 14:33

miltonhultgren commented Oct 17, 2024

View reviewed changes

kibanamachine and others added 2 commits October 17, 2024 15:14

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

4bd15fa

… src/core/server/integration_tests/ci_checks'

Merge branch 'main' into eem-non-hashed-ids

ada614c

miltonhultgren requested review from a team and removed request for a team and jaredburgettelastic October 18, 2024 07:47

jloleysens approved these changes Oct 18, 2024

View reviewed changes

klacabane approved these changes Oct 18, 2024

View reviewed changes

miltonhultgren merged commit ae2c6ad into elastic:main Oct 18, 2024
25 checks passed

kibanamachine added the v9.0.0 label Oct 18, 2024

kibanamachine mentioned this pull request Oct 18, 2024

[8.x] [EEM] Replace hashed ID with human readable ID (#193652) #196902

Merged

kibanamachine added the v8.17.0 label Oct 18, 2024

tiansivive mentioned this pull request Oct 31, 2024

[8.16] [EEM] Replace hashed ID with human readable ID (#193652) #198494

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EEM] Replace hashed ID with human readable ID #193652

[EEM] Replace hashed ID with human readable ID #193652

miltonhultgren commented Sep 21, 2024 •

edited by kibanamachine

Loading

obltmachine commented Sep 21, 2024

miltonhultgren commented Sep 21, 2024

dgieselaar Sep 30, 2024

klacabane Sep 30, 2024

miltonhultgren Oct 10, 2024 •

edited

Loading

dgieselaar Oct 10, 2024

miltonhultgren Oct 10, 2024

simianhacker Oct 10, 2024

kibana-ci commented Sep 30, 2024

miltonhultgren Oct 17, 2024

miltonhultgren Oct 17, 2024

elasticmachine commented Oct 18, 2024 •

edited

Loading

jloleysens left a comment

kibanamachine commented Oct 18, 2024

kibanamachine commented Oct 18, 2024

[EEM] Replace hashed ID with human readable ID #193652

[EEM] Replace hashed ID with human readable ID #193652

Conversation

miltonhultgren commented Sep 21, 2024 • edited by kibanamachine Loading

obltmachine commented Sep 21, 2024

🤖 GitHub comments

miltonhultgren commented Sep 21, 2024

dgieselaar Sep 30, 2024

Choose a reason for hiding this comment

klacabane Sep 30, 2024

Choose a reason for hiding this comment

miltonhultgren Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

dgieselaar Oct 10, 2024

Choose a reason for hiding this comment

miltonhultgren Oct 10, 2024

Choose a reason for hiding this comment

simianhacker Oct 10, 2024

Choose a reason for hiding this comment

kibana-ci commented Sep 30, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

History

miltonhultgren Oct 17, 2024

Choose a reason for hiding this comment

miltonhultgren Oct 17, 2024

Choose a reason for hiding this comment

elasticmachine commented Oct 18, 2024 • edited Loading

💚 Build Succeeded

Metrics [docs]

Async chunks

History

jloleysens left a comment

Choose a reason for hiding this comment

kibanamachine commented Oct 18, 2024

kibanamachine commented Oct 18, 2024

💚 All backports created successfully

Questions ?

miltonhultgren commented Sep 21, 2024 •

edited by kibanamachine

Loading

miltonhultgren Oct 10, 2024 •

edited

Loading

elasticmachine commented Oct 18, 2024 •

edited

Loading